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ABSTRACT 

"Design Inspection Reviews" are structured meetings in which 
participants follow certain rules of procedure and behavior when conducting 
detailed readings of design plans to identify errors and misunderstandings. 
The technique is widely used in the software engineering industry, where it 
is demonstrably more effective than testing at identifying errors in 
software, but it is by no means restricted to this domain. Similar practices 
can be found in many other creative industries. This paper reports on an 
interactive multimedia program used to reinforce the teaching of the 
technique to software engineering and computer network students. The results 
of the end-of-studies examination papers of the student group that used the 
multimedia program are compared to those of the previous group. The results 
appear to show an improvement of over 7% for the Review topic question for 
exams that were otherwise comparable in standard. The quality of the research 
data and the validity of it's conclusion are discussed. Includes two tables 
and three color figures. (Author) 
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Abstract: “Design Inspection Reviews” are structured meetings in which participants 
follow certain rules of procedure and behaviour when conducting detailed readings of 
design plans to identify errors and misunderstandings. The technique is widely used in the 
software engineering industry, where it is demonstrably more effective than testing at 
identifying errors in software, but it is by no means restricted to this domain. Similar 
practices can be found in many other creative industries. 

This paper reports on an interactive multimedia program used to reinforce the teaching of 
the technique to software engineering and computer network students. The results of the 
end-of-studies examination papers of the student group that used the multimedia program 
are compared to those of the previous group. The results appear to show an improvement 
of over 7% for the Review topic question for exams that were otherwise comparable in 
standard. The quality of the research data and the validity of it's conclusion are discussed. 



Introduction 

A common saying is “prevention is better than a cure”. In the industrial design environment this is certainly true. It is 
known to be far more cost effective to take extra time to identify problems at the design stage rather than try to work 
around the problems at the product installation or user operational stages. For example, this has been a central tenant 
of the quality assurance movement of the last two decades, as typified by the ISO 9000 standard. 

The truth of the saying has been known to the software development industry for many years. Fagan [1976] describes 
the application of Design Inspection Review techniques at IBM. Recent surveys of industrial practice show Design 
Reviews are today very widely used for .ensuring quality of software. Indeed [Love, 1999] found that the reviews are 
the top-most used approach for ensuring quality, more widely adopted than internal quality audits and statistical 
quality control procedures. 

Using lectures to teach the basics of Design Inspection Reviews can be ‘dry’ and for the student a rather passive 
experience. This paper describes the concepts and technologies of an interactive video that was developed to allow 
students to “chair” a Review meeting for themselves, giving individual viewers the opportunity to judge when the 
rules of the technique are being broken. The author makes observations on the success of the use of the interactive 
video, drawing upon data from examination results. However, these are only observations - the difficulties of drawing 
empirical conclusions are noted. 

Although the paper uses examples from software design to illustrate Design Inspection Reviews, the principles of the 
Review technique - and of multimedia program used to teach it - can be applied to many other situations. 



Design Inspection Reviews 

The premise of the Design Inspection Review technique is that the author / designer / originator is often too ‘close’ to 
their own piece of work be able to properly and objectively judge its fitness for purpose or technical qualities. Instead 
a small group of people assist the author to find all significant errors, omissions, inconsistencies and areas of 
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confusion in the product [Kelly, 1993]. Research has shown that an effective Inspection Review regime is more than 
paid back by reductions in the subsequent test-and-correct phase [Freedman, 1990]. 

The Design Inspection Review technique has three central tenants 

• The inspection team must be knowledgeable about both the product requirements and about the tools and 
methods used to design it. They must also have a vested interest in wanting to find problems before the product 
design is released. Typically this means that the team is comprised of direct peers of the author, plus the author 
him/herself. Fagan [1976, 1986] and Freedman & Weinberg [1990] recommend that anyone with line 
management responsibilities over the other team members should not be present as this could inhibit the open 
discussion of problems. Crawford-Hines [1996] dissents, pointing out that in industries such as software where 
seniority of position often is based on technical prowess, an immediate supervisor may have valuable expertise to 
share. 

• The Review is carried out according to schedule typically comprising of six phases: planning; participant 
briefing; individual preparation; inspection and defect recording; rework; and, if necessary, re-inspection. Each 
phase is structured towards the efficient use of time and towards the successful identification (rather than the 
solving) of problems. For example, the use of checklists of common errors is strongly recommended (together 
with a department-level process to review those lists to help Inspectors learn from other reviews about likely 
sources of problems). To ensure breadth of consideration, the team members are each allocated specific view- 
points from which to review the product: accuracy of working, ergonomics, and future maintainability, etc. 

• A number of conduct rules apply to the Inspection meeting itself. Two examples are:- 

In the months leading up to the Review the author will have invested many hours in their work. It represents, and 
reflects, their skills. At the Review meeting it is perhaps natural for the author to want to show to their colleagues 
f the complex or inventive parts of the solution. However, an author-lead discussion may not identify that the 
author either misunderstood or completely overlooked parts of the requirement specification. A rule of the 
meeting is that the Chair leads the discussion and the author just responds to questions. 

The product should only be inspected when the author believes it to be defect-free and ready for release. It can be 
a significant challenge to his/her ”ego” if the reviewers do then finds serious errors. The author may become 
defensive and attribute problems to others. Alternatively, they might even counter-attack and question the 
technical skills of the reviewer. In either case the effective, free-flowing discussion of potential problems would 
be lost. To prevent this the Chair of the meeting must ensure that all communication ~ verbal, and non-verbal too 
~ must suggest that “there is a problem with the product” and not “there is a problem with the producer”. 

This paper refers to the technique under the general title of “Design Inspection Review”. Over the years a number of 
variants - Fagan Inspections, Formal Technical Reviews, Active Design Reviews, Phased Inspections - have been 
proposed. Interested readers are referred to Freedman and Weinberg [1990] as the most comprehensive printed work. 
The Software Inspection and Review Organization web site has useful information though is a little dated [SIRO, 
1996]. The WWW Formal Technical Review Archive [FTRA, 2001] is a further source of online information. 
Trainers may find Powerpoint presentations of Design Inspection Reviews [Johnson, 1 998] useful. 



Previous approach to teaching Design Inspection Reviews 

Students on one of the undergraduate computing degrees at Sheffield Hallam University, England, are taught a 
version of Design Inspection Review based on Freedman & Weinberg. Students attend a lecture that describes the 
principles, and (prior to the development of a multimedia teaching tool) then attended classes in groups of around 
twenty where six volunteers would read aloud the dialogue of a “play” written by a course tutor. In the play the 
“characters” break many of the procedural and behavioural rules described in the lecture. All the students - the 
readers and the others - were encouraged take the perspective of the Chair of the meeting, and to interrupt when they 
identified transgressions. The tutor led the class in discussions as to the likely consequences. The class would then 
resuming with reading the script. 

These classes appeared to be popular with the students, with lively discussions in most sessions. However, there were 
some notable problems with the format: 
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• Tutors found it hard to get all students involved equally. Typically, a few noisy or confident students would 
dominate the discussions. 

• An educational aim was that the "play" should help inculcate in the students a sense of appropriate use of tone 
and language when making criticism. As with many issues of chairing a meeting, it is a matter of judgement 
knowing how far to let a discussion run before inteijecting. However, the first student to interject pre-empted the 
other students from making this judgement. 

• Students found it hard to revise the topic when preparing for end-of-year examinations. Perhaps because they 
were actively involved in reading and discussing the script, few took notes. Although the scripts themselves were 
available for aiding their revision, if a student did not pick up on a particular issue during the class session it is 
believed unlikely that they would spot it when re-reading the script weeks later. 



A modified approach to teaching: A multimedia program 

A multimedia program was created to address the problems identified above. A re-scripted version of the Design 
Inspection Review play was enacted, videoed, digitised and placed onto a CD, together with a media-controlling 
program to manage the interaction. The intent of the program was to permit individual students to take the role of 
“chair" of the meeting. They could interrupt the video whenever they thought that one of the Inspection rules was 
being broken and then use clickable buttons to identify the problem. The program would confirm the correctness or 
otherwise of the interruption, and give further feedback on the features of that class of problem. An important part of 
the design of the multimedia program was to guide students towards noticing problems that they had missed on the 
first viewing. 

The original concept for the program interaction was to create a ‘network’ of navigation paths through the video 
presentation. After students had interrupted the video and used the buttons and menus to identify what rule they 
thought was being broken, the program would supply a little further information (as text or extra video) about the 
consequences of this problem and where to get further information. It would then restart with a different video clip 
that showed a corrective action being taken. For example, if the viewer, in their role as “chair”, indicated that the 
language of criticism was too harsh then the next clip would start with an unseen voice (the “chair’s” voice) asking 
for more moderate tones. The subsequent clip would then progress towards a desired outcome. However, if the 
interjection was made at an inappropriate time the next clip would have a similar start, but the subsequent language 
might be so polite as to fail to uncover the next defect. In this design, not all the restart clips led to desirable 
outcomes. 

The network concept was not used in the final version of the program. It turned out that too many scripts were 
required for the time available to the project. It was also hard to find “natural” ways of giving feedback about the 
correctness of student interjections, especially if students selected faults that were not intended to be the primary 
problem at that point. A further problem was that the student might undermine the value of the package by navigating 
a route that missed all the important teaching points. Instead the program was designed to reflect that Inspection 
meetings have six sub-phases, and that most errors are most likely to occur in specific phases. For example, the 
attendance of an inappropriately skilled reviewer is likely to be detected at the meeting start-up phase. 

In the modified program design the student were presented with a predefined sequence of six video clips of between 
one and two minutes. They could interact to identify problems either as they came across them or at the end of the 
clip. If they interacted during a clip the video would be paused and feedback given as above, but on restart the same 
clip would be shown. Students could opt to see the clip again from its beginning, or from just before where they left 
off. To handle unexpected interjections, the machine would respond that “this problem can happen, but it isn’t 
happening here”. A central part of the interaction was that the program did not tell the viewer whether or not they 
had identified all the problems present in a clip. This was to encourage the student to be critical in their interactions 
rather than routinely identifying, say, two problems per scene, and simulates “real life”, where some sections of 
Design Inspection Meetings might run smoothly and require little correction, whilst others might require many 
interjections. 




Figure 2 - Screenprint of the program 

At the end of the whole program, the package lists which errors had been detected and which had been missed. 
Students then have the choice of watching again extracts of clips that contained identified or missed problems. The 
program concludes by giving a “score” based on the percentage of problems identified on the first viewing of the 
scenes. This “score”, though, is not considered a strong metric of performance (it takes no account of whether the 
problems was spotted during the showing of the clip or at the end, for example) and is not collected for evaluation 
purposes. 



Technical details of the creation of the Multimedia product 

A standard home VHS-S video camera was used to film a group of students role-playing an inspection meeting. 
Scenes from the video were played back through the camera and saved in AVI digital format by a computer (a 
Pentium II with 256Mb RAM and 4 Gbytes of unused disk space - a fairly standard specification machine, but 
equipped with a digital video capture card). Adobe Premier software was used to edit together different ‘takes’ of the 
same piece of script, either to remove errors and hesitations in the acting or to give different camera angles such as 
close-ups on individual speakers. The resulting sections of video film were saved in Real format. This editing and re- 
encoding into Real format reduced the storage size of the clips from several gigabytes to a few hundred megabytes. 

A computer program written in Macromedia’s Director language was used to control the playing of the video to a 
computer screen. A series of menu buttons were displayed beside the video (see figure 2). Director allows both the 
buttons and the actions taken when they are pressed to be varied by the program, so the buttons could be made to be 
appropriate to the content of the video being shown at the time. 

The video clips and program were written to CDs, which students could borrow to watch on a home or university 
machine. Alternatively they could download a copy across a network to watch on a university machine (downloading 
across a standard modem to a home machine would take prohibitively long). The two delivery formats gave identical 
results when watched. We did not try to stream the video because with current technologies there can be a significant 
delay - often around 20 seconds - before sufficient video is received to start the viewing. Eventually, advances in 
technologies may overcome this difficulty, and allow on-demand access to interactive learning materials without any 
administrative overheads. 

None of the above technical steps were found to be difficult. They were all within the leaning capability of the 
program’s author, a Final Year undergraduate Software Engineering student. It is noteworthy, too, how little attention 
needed to be paid to the filming environment. Acceptable results were gained using just the normal fluorescent lights 
of the meeting room and the camera’s built-in microphone. We did use a room with carpets to soften the sound and 
reduce the noise of chairs scraping. 
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Evaluation of the learning from the package 



Learning from the multimedia teaching tool, measured eight weeks after the use the video, was evaluated by repeating 
the same examination question given to the cohort prior to the use of the video onto the examination paper of the 
cohort that did have it available. This is shown as Q1 in the tables below. To check that any change in performance 
levels was not due just to question familiarity, a question from an unrelated topic was also repeated (Q2). The other 
questions were from different topics or gave a different emphasis to the same topic, and were not directly comparable. 
This is emphasised in the tables by labelling them Q3-Q7 and Qiii-Qvii respectively. The two papers each required 
the students to answer four questions from a choice of seven. 

The same group of tutors taught and examined the course, and the teaching approaches were broadly the same except 
for the topic of Inspection. The author of this paper was not a tutor on the course for either year, do direct 
experimenter influence is reduced. It is stressed, however, that still this was not intended to be a strict experiment and 
a number of important variables were not controlled. For example, students in the later year group had access to any 
new literature published during the later year. For this reason, this research paper makes observational findings based 
on data, it does not claim that these are empirically derived results. 

Tables 1 and 2 respectively show the examination results from the year before the use of the video and for the year of 
use. For each question they show the number of student answers, the question’s “popularity” (i.e. the percentage of 
students choosing to answer the question) and the mean of the marks given to the students’ responses. Similar data is 
shown for the exam overall, and for overall but excluding the Inspection question. To make the tables simpler to 
understand Q3-Q7 and Qiii-Qvii are presented in decreasing mean-score order. The questions did not appear in this 
order on the examination papers. 
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Table 1 Results for cohort previous to video 
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Table 2 Results for cohort that had video available 



Figure 1 : Comparison of results 



Tables 1 & 2 show that mean mark for the Inspection question (Ql) for the cohort before the video was 51 .2 and was 
58.7 after. It can also be seen that the mean mark for the "control" question (Q2) is only 0.2% different, suggesting 
that repeating a question does not itself explain the rise in mark for Ql. It can further be seen that the exam overall 
excluding the Inspection question is only 0.8% different between the two cohorts, suggesting that the two 
examinations were of more-or-less comparable levels of difficulty. Hence it might be concluded that the change in 
Ql's result from 5 1 .2 to 58.7 is due to the change in method of teaching, i.e. due to the introduction of the interactive 
video CD. Under the United Kingdom examination system, a 7.5 mark improvement represents not far off a whole 
degree-classification difference, and is a worthy achievement. 



(A less favourable interpretation could also be postulated: perhaps it was not the use of the video that lead to the 
improvement in answers to Ql, but that the previous use of the “play” so confused the students that their ’previous- 
cohort’ mark was actually depressed. The research does not have the evidence to refute this conclusion. It can only be 
stated that the professional judgement of the teaching team is that this was not the case.) 

An examination of the data was made to see if the profile of marks within the Inspection question had changed for 
those students who declared that they had watched the video. A total of 51 students out of 202 either told staff that 
they had downloaded a copy or signed as having borrowed a CD. Figure 2 presents a comparison of the exam results 
in graphical format. In the previous year the score for the Inspection question, at 51 .2, was nearly equal to the mean of 
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Figure 2 - Analysis of mean question results 



all the other questions. The Students who are known to 
have used the multimedia programme scored a mean of 
61.5 for the inspection question, an improvement of over 
10 marks. However, the “not watched” figure also raised 
by 5 marks, to 55.5. We consider it likely that many 
students watched the video but did not notify the staff, 
and the “not watched” column should really be labelled 
as "not reported as having watched the video". Although 
the graph shows that students that did watch the video 
did marginally better on their other questions (51.5) than 
the non-reporters (49.1), this is not considered to be a 
significant difference. 



Conclusions 

The “headline” conclusion of this paper is that eight weeks after teaching of a topic was supplemented by an 
interactive video, the marks for the relevant question on the examination paper were raised from 51.2 to 58.7 (and to 
61.5 for those who explicitly said that they had seen the video). This is after discounting factors such as experimenter 
influence, differences in the overall standard of the examination papers, or the re-use of a question leading to students 
being better prepared. 

The author is careful to state that whilst the above conclusion is based on numerical data this does not amount to 
empirical evidence, and has pointed out many factors that invalidate the results from being a correctly-formed 
“scientific experiment”. 

On a different aspect, a second finding of the research was the ease with which the interactive video was conceived 
and implemented. The researchers were surprised how few technical difficulties were met in the filming and digitising 
of the video, or in writing the presentational program. The largest difficulty was a pedagogic issue, namely of 
deciding the program’s content. 

The overall conclusion is that the apparent gain in retained learning of over 7% is noted, and that the project does 
merit the design of a carefully structured empirical study. 
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